feat: decompose patch 'other' error bucket into actionable sub-categories (Closes #1501) - #1502
Merged
Merged
Conversation
…ries Add old_string_empty and indentation_mismatch sub-categories to classify_error() and ERROR_TYPES in tools/fuzzy_match.py, with recovery hints in format_structured_error(). The indentation_mismatch detection uses a secondary classification pass that compares leading whitespace across corresponding lines when the primary classifier returns no_match — catching the common case where the model's old_string text matches but indentation (tabs vs spaces, indent level) differs. Update scripts/introspection_extract.py _classify_failure_reason() with regex patterns for the new sub-categories (patch-old-string-empty, patch-indentation-mismatch, patch-ambiguous), checked before the generic _NO_MATCH_RE so specific patch failures are no longer swallowed by the broad 'no match' or 'other' buckets. This shrinks the 68 failures/week 'other' bucket (87% of patch errors) by giving the model precise reasons and recovery paths. Closes #1501 Co-Authored-By: Hermes Evolution <evolution@hermes.ai>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Automated evolution PR for issue #1501.
Problem
The patch tool's
othererror bucket accounts for 68 failures/week (87% of all patch errors). These failures are classified asunknownbyclassify_error()andotherby_classify_failure_reason(), giving the model no actionable recovery path — leading to retry spirals on unknown causes.Solution
Decompose the
other/unknownbucket into actionable sub-categories:old_string_empty—classify_error()now recognizes"old_string cannot be empty"(the error fromfuzzy_find_and_replaceline 73) and classifies it asold_string_emptyinstead of falling through tounknown.indentation_mismatch— a secondary classification pass informat_structured_error()that triggers when the primary classifier returnsno_match. It finds the best-matching content line, then compares leading whitespace across ALL corresponding lines (the mismatch is often on a subsequent line, not the anchor). When the text matches but indentation differs (tabs vs spaces, different indent levels), it reclassifies asindentation_mismatch.Recovery hints — both new types get specific recovery suggestions:
old_string_emptytells the model to provide a non-empty old_string;indentation_mismatchtells it to useread_fileto see exact whitespace and update old_string to match.Introspection sub-categories —
_classify_failure_reason()inscripts/introspection_extract.pygets three new regex patterns (_PATCH_OLD_STRING_EMPTY_RE,_PATCH_INDENTATION_MISMATCH_RE,_PATCH_AMBIGUOUS_RE) checked BEFORE the generic_NO_MATCH_RE, so specific patch failures are taggedpatch-old-string-empty,patch-indentation-mismatch, orpatch-ambiguousinstead of falling intoother.Files changed
tools/fuzzy_match.py— new ERROR_TYPES entries, classify_error sub-categories, format_structured_error secondary classification + recovery hintsscripts/introspection_extract.py— new regex patterns + classification chain entriestests/tools/test_patch_self_correction.py— tests for new classify_error and format_structured_error pathsValidation
Closes #1501